Supply of smart devices such as smart phones and smart TVs have been making it active to study (or investigate) about user interface based on human gestures.
Currently, user interface methods based on user gestures include a contact sensing method which requires a user to directly touch a finger, etc. on a screen, and a non-contact sensing method which uses information acquired by using an imaging sensor, e.g., a camera sensor, without requiring the user to touch on the screen.
However, it is common that the existing user interface method using non-contact sensing methods simply develop a graphical user interface (GUI) that moves a medium like a mouse pointer on the screen and selects an icon by analyzing the user's motion.
Recently, three-dimensional (3D) display devices that provide 3D information for users such as 3D TVs are supplied.
To provide a more realistic interface environment for users by using the 3D display devices, it is necessary to provide a correct visual perception and reaction of virtual objects that makes a user feel as if he or she directly touch his or her hands on the perceived virtual objects, not by controlling a medium such as a pointer or an avatar. According to the conventional technology, a method for generating images for a user's correct visual perception by using an image sensor observing both the user and the images on the display simultaneously, is provided.
But it is usual to place the image sensor which is built in a smart phone, a smart TV, etc. in a same direction with a screen of such display device to observe a user or an object facing the screen so that the image sensor cannot observe the image displayed on the screen.
As such, under the conventional technology in the device environment where the image outputted through the screen cannot be directly observed by the image sensor, the conventional method is difficult to be applied and it is essential to set a location relationship between the display device and the image sensor that observes the user or the object.